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LONG-TERM  GOALS 

The  APL  atmospheric  sciences  group  is  working  to  improve  forecaster  performance  at  Navy 
operational  weather  forecast  detachments  afloat  and  ashore.  This  work  encompasses  broad  research 
and  technology  development  in  areas  of  visualization,  human  factors,  human-machine  interaction,  and 
model  and  forecast  verification  with  an  emphasis  on  mesoscale  ensembles  and  visualization  of 
uncertainty.  The  verification  effort’s  long-tenn  goal  is  to  develop  an  automated,  objective  verification 
technique  for  assessment  of  very  high-resolution  mesoscale  predictions  which  accurately  accounts  for 
spatially  or  temporally  misplaced  features,  false  alarms  and  misses  (Brown  2002). 

OBJECTIVES 

The  overall  objective  of  this  effort  is  to  develop  a  highly  automated,  rapid,  mesoscale  numerical 
weather  prediction  (NWP)  verification  tool  for  use  by  forecasters  and  model  developers.  The 
verification  technique  should  consider  distortion  errors  (phase/timing,  rotation,  and  stretching)  as  well 
as  the  normal  amplitude  errors.  It  is  intended  to  test  the  verification  tool  on  the  University  of 
Washington  Short  Range  Ensemble  Forecast  System  (SREF)  and  with  a  version  of  the  Navy  COAMPS 
model  to  be  implemented  at  APL. 

APPROACH 

The  verification  technique  of  Van  Galen  (1970)  and  Hoffman,  et.  ah,  (1995)  will  be  implemented  as  a 
rapid,  automated,  web-enabled  forecaster  and  model  developer  tool.  This  technique,  originally 
intended  for  synoptic-scale  features,  will  be  tested  on  mesoscale  predictions  of  a  number  of  parameters 
in  an  attempt  to  separate  model  forecast  errors  into  amplitude,  phase,  rotation,  and  distortion 
components  for  better  evaluation  of  mesoscale  model  capabilities.  In  order  to  improve  processing  time, 
image  motion  processing  techniques  (Chan,  1993;  Lim  and  Ho,  1998)  will  be  implemented  and  tested 
for  potential  acceleration  of  the  verification  routine.  The  verification  tool  will  be  used  to  evaluate  the 
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University  of  Washington  Short  Range  Mesoscale  Ensemble  (SREF)  performance  and  also  will  be 
tested  on  an  APL  version  of  COAMPS  when  implemented.  As  an  expansion  of  this  effort,  a  user- 
focused  cost-loss  evaluation  will  be  investigated  for  better  interpretation  of  component  verification 
(amplitude,  phase,  rotation,  distortion)  results.  Additional  participants  include  Dr.  Caren  Marzban 
added  to  the  effort  in  FY04  and  collaborations  with  Dr.  Cavan  Rielly,  U.  of  Minn,  and  Dr.  Jason 
Nachamkin  at  the  Naval  Research  Laboratory.  Our  group  has  been  approached  by  Dr.  Sandra  Yuter  of 
UW  for  applications  to  satellite  cal/val  and  by  Dr.  Craig  Bishop  of  NRL  for  development  of  model 
error  characteristics  for  data  assimilation.  Support  of  these  efforts  will  require  modification  of  code  to 
allow  it  to  easily  accommodate  disparate  data  sources.  In  order  to  deal  with  limitations  to  the  Hoffman 
technique  for  discontinuous  fields  such  as  precipitation  and  to  deal  with  multi-scale  processes,  a 
variety  of  statistical  techniques  are  being  reviewed  for  implementation  into  the  MVT  framework. 

These  include  agglomerative  cluster  analysis  (Marzban  and  Sandgathe,  2005)  and  variograms. 

WORK  COMPLETED 

FY2003  progress:  The  Hoffman  (1995)  technique  for  amplitude  and  phase  error  was  implemented  in 
both  MATLAB  and  web-enabled  versions  (see  figure  1  below)  and  fully  tested  on  idealized  and  real 
model  cases.  Three  image  motion  processing  techniques  were  implemented  in  the  routine  and  tested 
for  both  acceleration  and  accuracy.  Three  methods  of  error  minimization  (mean  absolute  difference 
(MAD),  route  mean  square  error  (RMSE)  and  mean  square  error  (MSE))  were  tested  both  for 
numerical  efficiency  and  accuracy  of  localization.  In  addition,  a  competing  verification  technique 
developed  by  University  of  Minnesota  was  tested  on  idealized  and  real  cases  and  a  paper  is  currently 
in  review. 

FY2004  progress:  A  combination  of  acceleration  techniques  developed  by  the  motion  picture  industry 
for  image  motion  processing  were  chosen  for  the  final  version  of  the  search  engine.  These  techniques, 
a  two-step  layered  structure  algorithm  and  an  inter-block  motion  algorithm,  resulted  in  approximately 
30x  acceleration  in  the  verification  routine  with  an  error  of  less  than  one  percent.  The  web-based 
verification  tool,  “Mesoscale  Verification  Tool”  or  MVT,  was  finalized  for  the  UW  ensemble  system 
and  implemented  on  the  UW  Mesoscale  Uncertainty  Monitor  (MUM)  for  use  by  local  Navy,  NWS  and 
other  forecasters  and  scientists.  A  web-based  batch  verification  tool,  the  “Automated  Mesoscale 
Verification  Tool”,  or  AMVT  was  implemented  for  rapidly  verifying  large  data  sets  including  multiple 
models,  multiple  parameters  and  long  time  series.  Both  MVT  and  AMVT  incorporate  the  search 
acceleration  techniques  that  allow  consideration  of  time  and  phase  errors  during  verification. 

Techniques  are  being  explored  to  deal  with  event-based  verification,  necessary  for  both  verification  of 
precipitation  which  is  discontinuous  in  areal  extent  and  for  verification  of  extreme  events  such  as  a 
frontal  passage.  Agglomerative  Hierarchical  Cluster  Analysis  (CA)  is  being  tested  as  a  means  to 
automatically  group  precipitation  observations  or  grid-based  predictions  into  related  ‘clusters’,  i.e., 
unique  precipitation  events,  to  allow  automated  verification.  Currently,  (Ebert  and  McBride,  2000  and 
Du  and  Mullen,  2000)  it  is  necessary  to  identify  precipitation  clusters  by  hand  analysis  prior  to 
verification  using  the  Hoffman  technique.  Using  CA  we  have  been  able  to  demonstrate  (Marzban  and 
Sandgathe,  2005)  a  potentially  automated  means  of  associating  observed  and  model  forecast 
precipitation  events.  Last  year  we  reported  the  selection  of  mean  square  error  (MSE)  as  the  primary 
minimization  parameter.  However,  we  have  found  that  for  extreme  events  such  as  frontal  passage,  the 
grid-based  Hoffman  technique  is  not  sufficiently  sensitive  to  capture,  uniquely,  the  phase  shift  of  a 
very  ‘sharp’  event  when  it  is  part  of  a  much  broader  field.  Consequently,  we  are  testing  the  use  of 
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mean  quadratic  error  (MQE)  as  the  minimization  parameter  in  our  search  routine.  The  selection  of 
MQE  over  MSE  highlights  the  most  extreme  values  and,  based  on  tests  to  date,  appears  to  easily 
capture  extreme  event  phase  shifts. 

FY05  progress:  The  MVT  and  AMVT  programs  have  been  recoded  for  efficient  and  more  user- 
oriented  operation.  The  software  has  also  been  expanded  to  allow  input  of  netCDF  files,  enabling  more 
flexibility  in  verifying  any  gridded  model  outputs  including  using  the  system  for  satellite  image 
calibration.  Portions  of  these  programs  are  being  transitioned  into  the  ONR  Visualization  of 
Uncertainty  Multidisciplinary  University  Research  Initiative  and  into  the  Environmental  Visualization 
(EVIS)  portion  of  the  ONR  Future  Naval  Capabilities  (FNC)  on  Knowledge  Superiority.  The 
remainder  to  the  CY05  effort  will  be  devoted  to  analysis  of  several  large  ensemble  case  studies  using 
AMVT  and  the  UW  SREF. 

A  cluster  analysis  technique  (Marzban  and  Sandgathe,  2005)  using  agglomerative  hierarchical  cluster 
analysis  which  incorporates  both  spatial  relations  and  parameter  value  (e.g.  precip  amount)  has  been 
tested  for  verification  of  discontinuous  fields  such  as  precipitation.  Figure  2  below  is  an  example  of 
CA  applied  to  a  precipitation  field.  This  technique  was  very  well  received  at  the  Montreal  Verification 
Workshop  (Marzban,  2004)  and  has  been  recently  funded  by  NSF  for  rapid  transition  into  the 
DoD/NOAA/NCAR  mesoscale  WRF  Developmental  Testbed  Center  (DTC). 

RESULTS 

FY03:  The  Van  Galen/Hoffman  technique  has  been  successfully  implemented  both  in  MATLAB  for 
further  scientific  investigation  and  as  a  prototype  web-based  forecaster  tool  in  the  UW  Mesoscale 
Uncertainty  Monitor  (MUM).  The  MUM  is  a  prototype  forecaster  aide  being  developed  for  testing  at 
NPMOD  Whidbey  Island.  An  approximately  30x  increase  in  processing  speed  in  the  verification  tool 
has  been  achieved  by  implementing  a  combination  of  two  image  motion  processing  routines  and 
optimizing  the  computer  code  functions.  In  addition,  it  was  determined  that  MSE  error  minimization 
for  localization  produced  the  most  reliable  and  accurate  verification  computation  without  significantly 
impacting  computational  speed. 

FY04:  The  Mesoscale  Verification  Tool  using  a  two-step  layered  structure  algorithm  and  an  inter¬ 
block  motion  algorithm  in  combination  has  been  transitioned  to  the  MUM  in  final  fonn.  The 
Automated  Mesoscale  Verification  Tool,  a  web-interactive  batch  verification  tool  has  been  developed 
and  tested  for  use  on  very  large  data  sets  with  multiple  parameters,  great  flexibility,  and  including 
multi-member  ensemble  verification.  A  mean  quadratic  error  algorithm  has  been  tested  and 
implemented  as  an  improved  search  criterion  for  extreme  event  verification  and  has  been  incorporated 
as  an  option  in  the  AMVT.  A  cluster  analysis  technique  using  agglomerative  hierarchical  cluster 
analysis  which  incorporates  both  spacial  relations  and  parameter  value  (i.e.  precip  amount)  has  been 
tested  for  verification  of  discontinuous  or  fields  such  as  precipitation. 

FY05:  The  AMVT  and  MVT  software  have  been  recoded  for  more  efficient  and  user  friendly 
operation  including  adding  the  capability  to  accept  netCDF  files.  This  capability  will  allow  input  of 
any  gridded  model  fields  for  verification  or  comparison  including  satellite  and  radar  image  calibration. 
The  agglomerative  cluster  analysis  technique  (Marzban  and  Sandgathe  2005)  has  been  selected  by 
NSF  for  incorporation  into  the  mesoscale  DTC. 
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IMPACT/APPLICATIONS 


Verification  systems  need  to  be  highly  automated  in  order  to  rapidly  assess  large  samples  of  cases; 
however,  they  must  also  be  able  to  correctly  evaluate  the  high  frequency,  high  amplitude  signals  of 
mesoscale  features.  Unfortunately,  traditional  methods  of  verification  have  been  shown  not  to  work  for 
mesoscale  numerical  weather  prediction.  Simple  automated  techniques  incorrectly  assess  slight  phase 
or  displacement  errors,  causing  smoothed,  or  ensemble  mean  forecasts  to  appear  to  perform  better  than 
a  more  detailed  deterministic  forecast,  yet  they  contain  less  forecast  content.  Case  studies,  while  more 
revealing,  are  too  time  consuming  to  assess  the  large  number  of  cases  required  for  subtle  model  biases 
or  differences  arising  from  small  changes  in  model  algorithms.  Mesoscale  NWP  forecast  verification  is 
a  critical  issue  for  US  Navy  operations.  More  NWP  model  outputs  are  becoming  available  from 
various  sources  and  it  is  difficult  for  management  and  operational  forecasters  to  choose  the  appropriate 
system  for  each  forecast  situation.  The  forecast  verification  tool  will  enable  more  accurate  and 
meaningful  evaluation  of  mesoscale  numerical  weather  prediction  systems,  especially  mesoscale 
ensemble  systems,  where  large  volumes  of  data  are  required  for  accurate  assessment  and  where  small 
prediction  distortions  or  displacements  cause  significant  misinterpretation  of  verification  results.  The 
tool  is  intended  for  use  both  by  model  developers  and  by  forecasters  for  quick  and  more  accurate 
model  assessment.  The  forecaster  tool  will  be  implemented  as  an  easy  to  use  web  tool. 

TRANSITIONS 

The  MVT  has  been  transitioned  to  the  MUM  or  Mesoscale  Uncertainty  Monitor  as  a  test  product.  It  is 
also  being  considered  by  the  Environmental  Visualization  (EVIS)  program  as  a  potential  web-based 
user  tool  and  by  the  NRL  as  a  COAMPS  On-scene  user  tool.  The  agglomerative  cluster  analysis 
technique  has  been  funded  by  NSF  for  transition  into  the  DoD/NCAR/NOAA  Mesoscale 
Developmental  Test  Center  (DTC). 

RELATED  PROJECTS 

The  University  of  Washington  Multidisciplinary  University  Research  Initiative  (MURI)  on  Integration 
and  Visualization  of  Multi-Source  Information  for  Mesoscale  Meteorology:  Statistical  and  Cognitive 
Approaches  to  Visualizing  Uncertainty.  This  project  incorporates  a  number  of  verification  techniques 
into  a  forecaster  visualization  tool  and  a  prototype  version  of  our  mesoscale  verification  tool  has  been 
implemented  here.  The  Environmental  Visualization  (EVIS)  effort  of  the  Knowledge  Superiority  FNC 
funded  by  ONR  has  reviewed  the  MVT  and  is  planning  to  test  it  as  a  potential  user  application. 
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Figure  1.  Automated  Mesoscale  Verification  Tool  GUI 
[Automated  Mesoscale  Verification  Tool  (AMVT)  Graphical  User  Interface  displaying  model  and 
parameter  selection  for  web-based  submission  of  large  and/or  complex  verification  of  multiple 

models  or  multi-member  ensembles.] 
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Figure  2.  Example  of  precipitation  field  decomposed  using  agglomerative  cluster  analysis. 
[The  field  on  the  left  is  a  plot  of precipitation  intensity  for  the  NE  Pacific  and  NW  America.  The 
field  on  the  right  is  the  same  field  indicating  clustering  of precipitation  based 

on  location  and  intensity.] 
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